Image capture input and projection output

ABSTRACT

Systems and methods are provided for alternating projection of content and capturing an image. The method includes steps of projecting, by a projection device, content at a first rate, projecting, by the projection device, a capture frame at a second rate, and capturing, by an image capture device, an image including the capture frame at the second rate, wherein capturing the image comprises capturing the image when the capture frame is projected. Systems and methods provided herein may provide increased tracking accuracy by a projecting a capture frame that does not obscure features in the image.

TECHNICAL FIELD

Embodiments disclosed herein are generally directed to systems capableof capturing an image and projecting content.

BACKGROUND

Augmented Reality (AR) is an emerging technology that allows a user tochange the way that they interact with the world. AR is a live, director indirect view of a physical, real-world environment wherein theelements are augmented by computer generated input such as sound, video,graphics, and/or Global Positioning System (GPS) data, for example. Insome uses, AR is enabled by a device that has an image capture devicethat captures images of the environment around a user and then usesobject recognition algorithms for recognizing objects in the capturedimages for augmentation. The user may then be provided with a view ofthe environment around the user on a display of the device with portionsof the view augmented based on the recognized objects, for example.

The development of personal, mobile, or pico projectors has providedadditional functionality to AR. For example, instead of or in additionto displaying a view of the augmented environment on a screen, a devicemay interact with the projector to recognize an object and projectcontent onto the actual object and/or onto an area near or around orassociated with the object, for example.

SUMMARY

Consistent with some embodiments, there is provided a method includingprojecting, by a projection device, content at a first rate, projecting,by the projection device, a capture frame at a second rate, capturing,by an image capture device, an image when the capture frame isprojected, and performing a computer vision function based at least inpart on the captured image. The method may also be embodied in anon-transitory computer-readable medium. The method may also be used tocreate an augmented reality.

Consistent with some embodiments, there is also provided a system. Thesystem includes a projection device configured to project content at afirst rate and a capture frame at a second rate. The system alsoincludes an image capture device configured to capture an image when theprojection device projects the capture frame. The system furtherincludes one or more processors coupled to the projection device and theimage capture device, the one or more processors configured to processthe captured image and generate the content or determine a location ororientation of the content for projection based at least in part on thecaptured image. The system further includes a memory coupled to the oneor more processors.

Consistent with some embodiments, there is further provided a system.The system includes means for projecting content at a first rate, meansfor projecting a capture frame at a second rate, means for capturing animage when the capture frame is projected, and means for performing acomputer vision function based at least in part on the captured image.

Consistent with some embodiments, there is also provided a methodincluding capturing an image including projected content, capturing animage including a projected capture frame, and performing a computervision function based at least in part on the image including aprojected capture frame. The capture frame may be projected at a firstrate. The image including the projected capture frame may be capturedbased on the first rate. In some embodiments, the content and thecapture frame may projected by a projection device. Moreover, the imageincluding the projected capture frame may be captured in response toreceiving a signal from the projection device or from a processingdevice coupled to the projection device, and/or the signal may bewirelessly received from a remote device. In some embodiments, themethod includes analyzing a captured image to determine whether thecaptured image comprises the image including the projected captureframe, and the determination may be based on a time stamp of thecaptured image or on identifying a fiducial or marker in the capturedimage. In some embodiments, the image including projected content mayinclude the image including the projected capture frame. In someembodiments, the content may be projected onto an object, wherein thecomputer vision function comprises recognizing or tracking the object

Consistent with some embodiments, there is further provided a methodincluding capturing at least one image of a field of view including anarea onto which content is being projected and onto which capture framesare being projected at a first rate, determining one or more images ofthe at least one image which includes at least a portion of a projectedcapture frame, and performing a computer vision function based at leastin part on the one or more images. In some embodiments, the content andthe capture frame may be projected by a projection device. Moreover, thedetermining may be based on a received signal and, in some embodiments,the signal is received from the projection device or from a processingdevice coupled to the projection device, or the signal may be wirelesslyreceived from a remote device. In some embodiments, the capturing may beperformed in response to the determining based on the received signal.In some embodiments, the determining may be based on the first rate andthe capturing may be performed in response to the determining based onthe first rate. In some embodiments, the determining may be based on atime stamp of the one or more images. Moreover, the determining may bebased on identifying a fiducial or marker in the one or more images. Insome embodiments, the capture frame may include content and one or morecapture areas. Furthermore, the content may be projected onto an object,and the computer vision function includes recognizing or tracking theobject.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a processing device, consistent withsome embodiments.

FIGS. 2A-2E illustrate an example of capturing images for input andprojecting content, consistent with some embodiments.

FIGS. 3A and 3B illustrate an example of capturing images for input andprojecting content, consistent with some embodiments.

FIGS. 4A and 4B illustrate another example of capturing images for inputand projecting content, consistent with some embodiments.

FIGS. 5A and 5B illustrate an example of capturing images for input andprojecting content in low-light situations, consistent with someembodiments.

FIG. 6 is a flowchart illustrating a process for capturing images forinput and projecting content, consistent with some embodiments.

FIG. 7 is a flowchart illustrating a process for projecting content andcapturing an image, consistent with some embodiments.

In the drawings, elements having the same designation have the same orsimilar functions.

DETAILED DESCRIPTION

In the following description specific details are set forth describingcertain embodiments. It will be apparent, however, to one skilled in theart that the disclosed embodiments may be practiced without some or allof these specific details. The specific embodiments presented are meantto be illustrative, but not limiting. One skilled in the art may realizeother material that, although not specifically described herein, iswithin the scope and spirit of this disclosure.

Portable or pico projectors, as described previously, may be used in ARsystems and video games to project content including images or otherinformation into an area, wherein the projected content may be based ona captured image. However, projected content may introduce additionallines, colors, and text onto objects in the field of view of the imagecapture device that may make it difficult for object recognition and/ortracking algorithms to continue to recognize and/or track the objects inthe field of view. Moreover, the projector may utilize objectrecognition and tracking algorithms to “track” the object to keep theprojected content in place as the projector and/or objects in the filedof view move, which can be hindered by the projection.

Accordingly, systems and methods for image capture input and projectionoutput that allow an image capture device to capture images forperforming computer vision functions including object recognition andtracking, event detection, motion estimation, object identification,object detection, object classification, and optical characterrecognition. The systems and methods may capture images during one ormore periods when the projector is not projecting content that canobscure the captured image. In particular, embodiments disclosed hereinmay capture an image when content is not being projected and when acapture frame is being projected.

FIG. 1 is a diagram illustrating a processing device 100, consistentwith some embodiments. Processing device 100 may be a mobile device suchas a smartphone, a tablet computer, a personal computer, a laptop ornetbooks, a set-top box (STB) such as provided by cable or satellitecontent providers, or a video game system console. Processing device 100may also be a head-mounted display (HMD) or other wearable computingdevice. In some embodiments, processing device 100 is implemented in anautomobile, for example in an entertainment center or console of anautomobile, or is included or implemented in a healthcare device.According to some embodiments, processing device 100 may be implementedusing any appropriate combination of hardware and/or software configuredfor capturing images and projecting content. In particular, processingdevice 100 may include any appropriate combination of hardware and/orsoftware having one or more processors and capable of readinginstructions stored on a non-transitory machine-readable medium forexecution by the one or more processors for capturing images andprojecting content. Some common forms of machine-readable mediaincludes, for example, floppy disk, flexible disk, hard disk, magnetictape, any other magnetic medium, CD-ROM, any other optical medium, punchcards, paper tape, any other physical medium with patterns of holes,RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge,and/or any other medium from which one or more processors or computer isadapted to read.

Processing device 100 may include network interface component 102configured for communication with a network. Consistent with someembodiments, network interface component 102 may be configured tointerface with a coaxial cable, a fiber optic cable, a digitalsubscriber line (DSL) modem, a public switched telephone network (PSTN)modem, an Ethernet device, and/or various other types of wired networkcommunication devices. Network interface component 102 may also includeone or more wireless transceivers, wherein each wireless transceiver mayinclude an antenna that is separable or integral and is capable oftransmitting and receiving information according to a different wirelessnetworking technologies, such as Wi-Fi™, 3G, 4G, HSDPA, LTE, RF, NFC.

Consistent with some embodiments, processing device 100 includes asystem bus 104 for interconnecting various components within processingdevice 100 and communicating information between the various components.In some embodiments, the bus 104 is implemented in a System on Chip(SoC) and connects various elements or components on the chip and/orcores of one or more processors. Components may include a processingcomponent 106, which may be one or more processors, central processingunits (CPUs), image signal processors (ISPs), micro-controllers, ordigital signal processors (DSPs), graphics processing units (GPUs), andaudio signal processors, which may include analog and/or digital audiosignal processors. Components may also include a memory component 108,which may correspond to random access memory (RAM), read only memory(ROM), optical, magnetic, solid-state, or other memories such asdescribed above.

Consistent with some embodiments, processing device 100 may also includea display component 110 for displaying information to a user. Displaycomponent 110 may be a liquid crystal display (LCD) screen, an organiclight emitting diode (OLED) screen (including active matrix AMOLEDscreens), an LED screen, a plasma display, or a cathode ray tube (CRT)display. Display component 110 may be integrated with processing device100, or may be separate from processing device 100 and coupled toprocessing device 100. Processing device 100 may also include an inputand navigation control component 112, allowing for a user to inputinformation and navigate along display component 110. An input andnavigation component 112 may include, for example, a keyboard or keypad, whether physical or virtual, a mouse, a trackball, or other suchdevice, or a capacitive sensor-based touch screen.

Processing device 100 may also include sensors 114 that capture dataassociated with device 100 and/or its surroundings. Sensors 114 mayinclude, but are not limited to, microphones or audio sensors, cameras,light sensors, proximity sensors, depth sensors, pressure sensors,inertial sensors (e.g., accelerometers and/or gyroscopes),magnetometers, etc. Sensors 114 may be used individually or incombinations, such as sensor arrays or any other combinations. Sensors114 are capable of operating interdependently or independently of oneanother. Sensors 114 may communicate with processing component 106through system bus 104 in order to process data captured by sensors 114consistent with instructions stored in, for example, memory component108 and other modules, and to generate or otherwise obtain metadataassociated with captured data.

Processing device 100 may also include an image capture device 116. Insome embodiments, image capture device 116 may be a camera, such as avisible light camera or a depth-sensing camera, such as the Microsoft®Xbox™ Kinect™ camera. Image capture device 116 may also be configured todetect infrared (IR) light or ultraviolet (UV) light. Image capturedevice 116 may also be a stereo camera, a time-of-flight (ToF) camera,or other camera capable of detecting a capturing an image. In someembodiments, image capture device 116 may be configured to capture oneor more images of objects within the field of view of image capturedevice 116 for processing. In some embodiments, the image capture device116 is included in the sensors 114.

Processing device 100 may also include a projection device 118.Projection device 118 may be a device capable of projecting light toform one or more images. The one or more images may include content suchas static or moving images that are projected as a collection of frames.In some embodiments, projection device 118 may alter the placement ofthe projected light to give a projected image the appearance of movementor animation. Projection device 118 may be a Digital Light Processing(DLP) projector, a laser beam-steering (LBS) projector, a liquid crystalon silicon (LCOS) projector, or other projector. In some embodiments,projection device 118 may also be a mobile or portable projector.

Processing device 100 may also be capable of projection generation 120,and computer vision functions such as object recognition 122 andtracking 124. These functions may be performed by one or more softwaremodules that perform a function when executed by processing component106. In other embodiments, the functions may refer to an ApplicationSpecific Integrated Circuit (ASIC) or other circuit having memory and atleast one processor for executing instructions to perform the function.Thus, the functions 120-124 may be separate from other components in thedevice 100, as illustrated in FIG. 1, or may be combined and/orimplemented in one or more other modules illustrated in FIG. 1, forexample in the processing component 106. In some embodiments, projectiongeneration 120 may include generating content including one or moreimages for projection by projection device 118 onto a target object oran area for viewing, interacting, and/or other consumption by one ormore users of device 100 and/or others who may not be using device 100but may be within a viewable distance of the target object or area. Insome embodiments, the content may be content for augmenting a targetobject or area.

In some embodiments, projection generation 120 may also includegenerating a capture frame for projection by projection device 118. Thecapture frame may be a frame that is projected at a frame rate that islower than the frame rate of the projected content and may bealternatingly projected with the projected content or projected alongwith the projected content. The capture frame may be projected to allowimage capture device 116 to capture an image without projected content,so that the projected content does not obscure features within thecaptured image for processing. In some embodiments, the projectedcapture frame is projected at a frame rate that is sufficient for objectrecognition 122 or tracking 124 while remaining unnoticed by anobserver. In some embodiments, content may be projected by projectingdevice at a frame rate of about 23.976-300 frames per second (fps),while a capture frame is projected at a rate of about 10-20 fps. In someembodiments, content may be projected at a frame rate of about 72 fpsand the capture frame may be projected at a rate of about 15 fps. Insome embodiments, the projection rates of content and the capture framemay be higher or lower depending on capabilities of image capture device116, projection device 118, and/or the needs of a particular applicationor experience. In some embodiments, image capture device 116 may beconfigured to capture images at a predetermined rate that is independentof the projection of a capture frame, wherein object recognition 122 andtracking 124 is performed on those captured images that include captureframes. In other embodiments, image capture device 116 may be configuredto capture images at a rate that is related to the projection of thecapture frame. For example, image capture device 116 may be configuredto capture images at a rate that is at least two times the projectionrate of a capture frame such that at least one image is captured whenthe capture frame is projected and at least one image is captured whenthe capture frame is not projected.

In some embodiments, the capture frame may be a blank frame where nocontent is projected. In other embodiments, the capture frame may be aframe of a solid neutral color that does not have a pattern, or has avery subtle and minimal pattern that is projected to allow image capturedevice to capture an image that does not obscure lines, features, orboundaries of objects within the field of view of image capture device116 that may be processed for tracking 124, object recognition 122,and/or other computer vision functions. In some embodiments, the solidneutral frame of the solid neutral color may have a very light color,such as white, a color with very low saturation, such as gray, or a verydark color, such as black. Moreover, the capture frame may include or becomposed of a frame of high light intensity that may be used toilluminate the field of view to facilitate image capture by imagecapture device 116 in low-light situations. In some embodiments, theintensity is determined by an ambient light sensor and/or informationfrom the image capture device 116. In some embodiments, the captureframe may include a marker, fiducial, or other indicator that, whencaptured by image capture device 116 and processed, provides anindication that the captured frame with the marker or fiducial is theframe to be used for performing computer vision functions, such asobject recognition 124 and tracking 122. Further, in some embodiments,the capture frame may be projected along with content and may be aportion or subset of the projected content which includes no content orno images.

In some embodiments, projection device 118 and image capture device 116may be synchronized so that image capture device 116 captures an imageof an area in the field of view when projection device 118 is projectinga capture frame. In some embodiments, projection device 118 may becapable of sending a signal to image capture device 116 to indicate thata capture frame is being projected. The signal may be transmitted viasoftware, or may be sent over a hardware line. The signal may also besent wirelessly from a remote device, for example when distributedcomponents are implemented, some embodiments of which are describedbelow. Alternatively, the signal may be provided along with thegeneration of the capture frame in projection generation 120. Moreover,projection device 118 and image capture device 116 may be synchronizedusing clocking provided by clock signal generator 126 so that projectiondevice 118 and image capture device 116 may be capable of simultaneouslyprojecting a capture frame and capturing an image. In some embodiments,the synchronized clocks of projection device 118 and image capturedevice 116 may allow projection device 118 and image capture device 116to have aligned duty cycles, for example allowing in some embodimentsfor image capture device 116 to capture images based on a projectionrate of a capture frame, content, or a combination thereof.

Object recognition 122 may include performing one or more objectrecognition algorithms on images captured by image capture device 116.The object recognition algorithms performed in object recognition 122may be capable of recognizing an object in one or more image framesusing feature detection, pattern recognition, appearance matching, imagematching, edge and/or line detection, and the like. In some embodiments,objects recognized in object recognition 122 may be objects used fortracking 124, objects designated as targets of a projection, and/orobjects for augmentation. In some embodiments, object recognition 122may include automatically recognizing and designating recognized objectsas targets for projection and/or augmentation.

In some embodiments, a user of processing device 100 may be capable ofdesignating an object within a field of view as a target for projectionor an object for augmentation by selecting or otherwise interacting withthe object. In some embodiments, the object may itself be manipulated bythe user, while in other embodiments the user may interact with theobject as it is displayed on processing device 100 by, for example,display component 110. If a user selects an object for augmentation,object recognition 122 may include attempting to recognize the selectedobject and augment the recognized object with content based oninformation provided by a third party or by information related to theobject. In some embodiments, object recognition 122 may includecommunicating with a network using network interface component 102 toperform object recognition 122 and to find content related to theobject. In some embodiments, recognition of certain objects issufficient to determine an augmentation without action or input by theuser.

Tracking 124 may include associating a location of objects within atarget area over time. In some embodiments, tracking 124 may beconfigured to control a projection based on the determined location andorientation of objects in a target area over a period of time so thatthe projected content remains projected on one or more designated targetobjects even as the projector moves or the target objects move. Tracking124 may control a projection of projection device 118 based on imagescaptured by image capture device 116 and, in some embodiments,information from sensors 114. In some embodiments, tracking 124 mayinclude estimating pose, orientation, and depth information about targetobjects using images captured by image capture device 116 and/orinformation obtained by sensors 114, such as an accelerometer orgyroscope. Tracking 124 may further include generating one or morematrices that may later be used in projection generation 120 forcontrolling a projection by projection device 118, including generatingaugmented reality content. Tracking 124 may include scaling, rotatingand/or translating content for projection generation 120 based oncaptured images from image capture device 116 and/or information fromsensors 114. Tracking 124 may include determining six degrees-of-freedombased on the captured image and tracking content for projection based onthe determined six degrees-of-freedom. For embodiments where imagecapture device 116 is or includes a depth sensing camera, the one ormore images may be processed to develop a depth map of the field ofview. The depth map may be used in tracking 124 for tracking andcontrolling the projection, including maintaining a proper focus of theprojection.

Processing device 100 may also include a clock signal generator 126.Clock signal generator 126 may be configured to generate a clock signalused by components of processing device 100. In some embodiments, imagecapture device 116 may be configured to capture an image when projectiondevice 118 is projecting a capture frame based on a predetermined timingcorresponding to a projection rate of the image capture frame, which maybe projected at a rate of 10-20 fps, as discussed above. For example,image capture device 116 may be configured to capture an image of thefield of view that includes the projected capture frame every 67 msbased on a projection rate of an image capture frame of 15 fps, thetiming being provided to image capture device 116 by a clock signalgenerated by clock signal generator 126. Similarly, projection device118 may be configured to project a capture frame every 67 ms consistentwith an image capture frame projection rate of 15 fps based on a timingprovided by the generated clock signal. Alternatively, image capturedevice 116 may be configured to capture an image at a rate that is atleast two times the projection rate of the capture frame such that atleast one image is captured when the capture frame is being projectedand at least one image is captured when the capture frame is notprojected.

In embodiments wherein image capture device 116 is configured to captureimages at a rate independent of the projection rate of the captureframe, images not including a capture frame, which may include captureframes having a fiducial or marker, may be discarded or given a lowweight during processing based on the known rate of projection of thecapture frame or by clock signals associated with the captured frames.Projecting a fiducial or marker may reduce the coordination implementedbetween the projection device 118 and image capture device 116 in someembodiments. Moreover, clock signal generator 126 may be used byprocessing device 100 to timestamp captured images, with the timestampsof capture frames being known and used to know which captured images toanalyze and/or perform object recognition 122, tracking 124, and othercomputer vision functions on.

Processing device 100 may include more or less components than shown inFIG. 1 according to some embodiments. Moreover, components shown in FIG.1 may be directly coupled to one or more other components in FIG. 1,eliminating a need for system bus 104. Furthermore, components shown inFIG. 1 may be shown as being part of a unitary device 100, but may alsobe part of a system where the components are separate but coupled and incommunication. In general, the components shown in FIG. 1 are shown asexamples of components in a processing device 100 capable of performingembodiments disclosed herein. However, a processing device 100 may havemore or fewer components and still be capable of performing someembodiments disclosed herein.

FIGS. 2A-2E illustrate an example of capturing images for input andprojecting content, consistent with some embodiments. For the purpose ofillustration, only a portion of processing device 100 is shown in FIGS.2A-2E; however reference may be made to other components of processingdevice 100 shown in FIG. 1. As shown in FIG. 2A, image capture device116 may be capable of capturing an initial image of a field of view thatincludes object 1 200. In some embodiments, a computer vision functionsuch as object recognition 122 may be performed on the captured image toattempt to recognize object 1 200. In some embodiments, a pose may bedetermined from an initial image of a field view that includes object 1200. In some embodiments, the captured image may be a subsequentcaptured image after capturing an initial image and tracking 124 mayalso be performed on the captured image to, for example, determine apose, position, six degrees-of-freedom (DOF), and/or other informationabout object 1 200. In some embodiments, image capture device 116 mayinclude a depth camera which may be used to determine a depth map of thefield of view including object 1 200, which may be used for tracking 124of object 1 200 and for focusing a projection on object 1 200.

As shown in FIG. 2B, projection device 118 may project content 202 intothe field of view, including on and around object 1 200. In someembodiments, the projection of content 202 onto and around object 1 maybe controlled based on tracking 124 subsequent captured images, asdescribed above. In some embodiments, the alignment and/or position ofprojected content may be adjusted based on tracking object 1. Moreover,content 202 may also be controlled based on information obtained fromsensors 114, which may include an orientation of device 100 and/orprojection device 118, whether device 100 and/or projection device 118is stable, and a brightness level of the field of view. In someembodiments, content 202 may be generated, in part or in whole in someembodiments, based on object recognition 122. Object recognition 122 mayinclude identifying features of object 1 200, and searching to find amatching object and content 202 associated with the matching object. Thesearching may be performed based on features representative of objectsin memory component 108, or over a network. A matching object may haveassociated content 202 that may be projected on and around object 1 200.The associated content 202 may include information related to object 1200 such as features of object 1 200 or instructions for using object 1200. In some embodiments, content 202 associated with a matching objectmay be used to augment object 1 200 and an area around object 1 200 tocreate an augmented reality around object 1 200.

In some embodiments, content 202 may also include user selectedfeatures. For example, an image of object 1 200 captured by imagecapture device 116 may be displayed on display component 110 of device100 and a user of device 100 may be capable of interacting with thedisplayed image using input and navigation control component 112 toselect or create features that may be included in content 202. Thesefeatures may be included along with content from object recognition 122in projection generation 120 of content 202. Content 202, which mayrefer to elements within a projection or to the entire projectionprojected from projection device in some embodiments, may also includeone or more specific images, such as image 1 204 and image 2 206. Asshown in FIG. 2B, content 202 may include image 1 204 projected onobject 1 200 image 2 206 projected below object 1 200. In someembodiments, the projection of content 202 including image 1 204 andimage 2 206 on object 1 200 may augment object 1 200 to create anaugmented reality around object 1 200.

As shown in FIG. 2C, object 1 200 is located a distance d away from itsprevious position. In order to capture this change in position fortracking 124 and to determine additional information about the field ofview, projection generation 120 may include periodically generating acapture frame 208 for projection by projection device 118. In someembodiments, capture frame 208 may be a blank frame where no content isprojected. Capture frame 208 may also be a frame of a solid neutralcolor that does not obscure lines, features, or boundaries of object 1200. As capture frame 208 is projected by projection device 118, imagecapture device 116 may be capable of capturing an image of a field ofview 210 including capture frame 208. Although capture frame 208 isshown as being projected to cover field of view 210 of image capturedevice 116, the dimensions of capture frame 208 may be larger orsmaller, and may be set by a user, manufacturer, or developer and couldvary in size in relation to the projected content 202 and/or object 1200.

In some embodiments, projection device 118 may be capable of sending asignal to image capture device 116 over bus 104 when capture frame 208is being projected. In some embodiments, image capture device 116 andprojection device 118 may be synchronized based on a clock signalgenerated by clock signal generator 126 so that capture frame 208 isprojected and an image of field of view 210 are simultaneously capturedat a predetermined timing. In some embodiments, this predeterminedtiming may be sufficient to enable tracking 124 while capture frame 208remains unnoticed by a human eye. In some embodiments, image capturedevice 116 and projection device 118 may be synchronized based on aclock signal generated by clock signal generator 126 such that imagecapture device 116 captures at least one image of field of view 210 thatincludes capture frame 208 and at least one image of field of view 210that does not include capture frame 208.

As shown in FIG. 2D, after capture frame 208 is projected, content 202including image 1 204 and image 2 206 may again be projected byprojection device 118 on and/or around object 1 200. Based on tracking124 and other processing performed on the captured image of field ofview 210 when capture frame 208 was projected, the position andorientation of content 202 including image 1 204 and image 2 206 may becorrected based on the displacement d of object 1 200 with respect toits previous position.

In some embodiments, content including image 1 204 and image 2 206 mayhave changed and may have dimensions that are larger or smaller thanimage 1 204 and image 2 206. In order to maintain the position andorientation of content 202 including any changed images, the content 202or any elements thereof may be scaled, positioned, and/or otherwisemanipulated such that the position and/or orientation of contentprojected onto and around object 1 200 remains constant.

In some embodiments, processing device 100 may be able to trim a portionof content and/or create a mask that allows content 202 to be projectedaround an object while a capture area 212 is projected on an expectedposition of an object determined through, for example, tracking 124. Asshown in FIG. 2E, for example, projection device 118 projects a captureframe having content 202 including image 2 206 around object 1 200 whileprojecting capture area 212 around an expected position of object 1 200.Capture area 212 may be an area where no content is projected, and maybe blank or a neutral color. In some embodiments, image capture device116 may be configured to capture an image of an area including capturearea 212. In some embodiments, capture area 212 may include a fiducialor marker and image capture device 116 captures an image of an areaincluding content 202 and capture area but object recognition 122 and/ortracking 124 may only be performed on a captured area defined withrespect to the fiducial or marker, or the fiducial or marker mayindicate that the projection includes the capture area 212. In some suchembodiments, the fiducial or marker may be hidden within the content 202such that the user is not aware of the fiducial or maker and/or todecrease the likelihood of the image or marker obscuring object 1 200.In some embodiments, the processing device 100 interleaves captureframes having content and capture frames which do not include content.Those of skill in the art will appreciate that while one capture area isillustrated in FIG. 2E, additional capture areas may be included in thecapture frame, for example when tracking a plurality of objects (e.g.,as shown in some embodiments pursuant to FIGS. 3-4).

In one example, object 1 200 may be a printer or other device andcontent 202 may be instructions for operating the printer or otherdevice. Image 1 204 and image 2 206 may be positioned on specificbuttons, displays, or other parts of the printer or other device and mayinclude instructions relevant to the specific area. For example, ifobject 1 200 is a printer that includes a display that is displaying anerror message, object recognition 122 of a captured image may recognizethe printer type and the error, and projection generation 120 mayinclude generating content 202 and images 204 and 206 that includeinstructions for fixing the error.

Although processing device 100 is shown FIGS. 2A-2E as being a singledevice, in some embodiments processing device 100 may have separatedistributed components or may omit one or more elements that areimplemented in another device or system. For example, image capturedevice 116 may be located separate from projection device 118. In somesuch embodiments, both image capture device 116 and projection device118 are in communication with processing device 100. For example, imagecapture device 116 may be an image capture device associated with avideo game console, and may be configured to recognize and/or track auser or other object or may transmit information to the processingdevice 100 to recognize and/or track, for example using recognition 122and/or tracking 124. Projection device 118 may be configured to projectcontent on and around the user or object to create a (e.g., immersive)game experience for the user. Further, processing device 100 may becapable of detecting gestures made by the user to control aspects of thevideo game or other features of the projected content. In suchembodiments, projection device 118 may be capable of projecting acapture frame 208 or capture area 212 at a second rate such thatgestures made by the user may be observed or such that content may bemoved or adjusted based on images of the user captured when captureframe or capture area 212 is projected. In some embodiments, the imagecapture device 116 is implemented in a remote or control device, forexample as may be used with a displayed or projected content or with agaming system. Any of the techniques used to time capture of an image orindicate which frame comprises a capture frame or includes a capturearea described above may be implemented when distributed components areutilized. In some embodiments, one or both of the image capture device116 and projection device 118 are included in a wearable device, forexample a head mounted display (HMD), a device worn around the wrist orarm of a user, a device configured to hang about the neck of the user,smart fabric or clothing, and/or other wearable elements. In someembodiments, one or more elements may be implemented in a system ordevice that is independent of the other elements described with respectto the processing device 100. For example, in some embodiments, theprojection device 118 may be omitted and independent from the processingdevice 100. In an example, the processing device 100 may comprise amobile device such as a smartphone and may capture and/or process imagescomprising a capture frame projected by the independent projectiondevice 118. In one example, the projection device 118 may projectcontent associated with an application (e.g., “app”) utilized by themobile device. The application may be configured to identify captureframes or areas, for example based on a known or determined fiducial ormarker, and/or may present information to the user regarding an objecton which content is projected. Those of skill in the art will appreciatethat the distributed and/or independent components described above maybe implemented in and/or used with any of the embodiments describedherein, for example with one or more of the embodiments described withrespect to FIGS. 3-7.

FIGS. 3A and 3B illustrate an example of alternating capturing an imageand projecting content, consistent with some embodiments. For thepurpose of illustration, only a portion of processing device 100 isshown in FIGS. 3A and 3B, however reference may be made to othercomponents of processing device 100 shown in FIG. 1. Moreover, referencemay be made to FIGS. 2A-2E for illustration. As shown in FIG. 3A, asecond object, object 2 300 is now in the field of view of image capturedevice 116 along with object 1 200, which in some embodiments may be thesame object 1 200 from FIGS. 2A-2E but, in other embodiments, may be adifferent object. In some embodiments, the object 1 200 is the same asin FIG. 2B and has been introduced subsequent to the configurationillustrated therein. As capture frame 208 is periodically projected byprojection device 118, object 2 300 may be identified, and featuresassociated with object 2 300 may be processed for object recognition 122and tracking 124. Moreover, because capture frame 208 may be a blankframe where no content is projected or a frame of a solid neutral colorthat does not obscure lines, features, or boundaries of object 1 200 andobject 2 300, processing features and boundaries associated with object1 200 and object 2 300 for object recognition 122 and tracking 124 maybe improved.

As shown in FIG. 3B, based on processing the image captured by imagecapture device 116 while capture frame 208 was being projected,projection generation 120 may include content 302 that may be related toboth object 1 200 and object 2 300. Moreover, content 302 may beprojected on and/or around object 1 200 and object 2 300 based on aposition of both object 1 200 and object 2 300 as determined byprocessing the image captured by image capture device 116 as captureframe 208 was projected for tracking 124. For example, content 302 mayinclude image 1 304 for projection onto object 1 200 at a particularlocation and image 2 306 for projection onto object 2 300 at aparticular location. In some embodiments, the content and position ofimage 1 304 and image 2 306 may be designated by a user of device 100.In some embodiments, the content and position of image 1 304 and image 2306 may be predetermined based on object 1 200 and object 2 300 based onobject recognition 122. For example, the content and position of image 1304 and image 2 306 may be set by a manufacturer, distributor, ormerchant of object 1 200 or object 2 300, a content provider, or an appdeveloper.

In one example, object 1 200 and object 2 300 may be playing or tradingcards. Object recognition 122 may recognize object 1 200 as a particularcard, tracking may determine a position of the card, and projectiongeneration 120 may include generating content related to that card, suchas the name, value, and other information related to that card that maybe projected on or around the card. When object 2 300 is added intofield of view 210, object recognition 122 may recognize object 2 300based on an image captured as image capture frame 208 is projected asanother card, and tracking 124 may determine a location of the card.Projection generation 120 may then include generating content related tothe second card as well as the first card, the content being projectedonto and around each card. Separate content may be projected for eachcard or related content may be projected for both cards. In someembodiments, the content being projected onto and around each card maybe unified content that is projected around both cards. The unifiedcontent may illustrate a dialog or confrontation between charactersprinted on the cards. In some embodiments, the processing device 100 maydetermine if content for one of the cards has priority and may projectcontent appropriately. For example, only the content having priority maybe projected, or the content having priority may be projected over othercontent.

FIGS. 4A and 4B illustrate another example of alternating capturing animage and projecting content, consistent with some embodiments. As shownin FIG. 4A, the position of object 1 200 and object 2 300 may havechanged with respect to the position shown in FIG. 3B. In someembodiments, when projection device 118 projects capture frame 208,image capture device 116 may capture an image that may be used forprocessing. The captured image may be used for performing computervision functions such as tracking 124 and object recognition 122, forexample. Processing object 1 200 and object 2 300 with tracking 124 mayreveal the new positioning of object 1 200 and/or object 2 300 withrespect to the position of object 1 200 and/or object 2 300 previouslyshown in FIG. 3B. As shown in FIG. 4B, the positioning informationdetermined by tracking 124 may be used for projection generation 120such that content 302 including image 1 304 and image 2 306 is projectedon and around object 1 200 and object 2 300 based on the new position ofobject 1 200 and object 2 300. Those of skill in the art will appreciatethat while two objects are illustrated in FIGS. 3-4, additional objectsmay be recognized and/or tracked, or some other computer vision functionmay be performed with respect to such additional objects. Further, thoseof skill in the art will appreciate that the processing device 100 mayomit recognition 122 and/or tracking 124 while processing one or morecapture frames.

FIGS. 5A and 5B illustrate an example of alternating capturing an imageand projecting content in low-light situations, consistent with someembodiments. As shown in FIG. 5A, object 1 200 may be in an area 500having low light. Because image capture device 116 may use a relativelyfast shutter speed and short exposure time, images captured of object 1200 in area 500 may have low or poor image quality in someimplementations. As a result, features, boundaries, and otherinformation about object 1 200 and area 500 may not be easily detectedfor object recognition 122 or tracking 124. Any projection based onprocessing an image having poor image quality may also have poor qualityor may be incomplete or incorrect for object 1 200.

As shown in FIG. 5B, projection generation 120 may include generating acapture frame 502 having a high-light intensity that may betterilluminate the portion of area 500 onto which capture frame 502 isprojected. In some embodiments, image capture device 116 may be capableof capturing an image of a field of view 504 including capture frame 502having the high-light intensity. The captured image may then beprocessed for object recognition 122 and/or tracking 124, and mayprovide improved results as features, boundaries, and lines of object 1200 and field of view 504 are better illuminated. In some embodiments,capture frame 502 may be a frame of a solid color having high lightintensity. In some embodiments, capture frame 502 may be a blank frameand processing device 100 may include a flash lamp or light emittingdiode (LED) or the like that may be activated or illuminated whencapture frame 502 is projected. In some embodiments, projection device118 may be capable of sending a signal to image capture device 116 overbus 104 when capture frame 502 is being projected. In some embodiments,image capture device 116 and projection device 118 may be synchronizedbased on a clock signal generated by clock signal generator 126 so thatcapture frame 502 is projected and an image of field of view 504 iscaptured at a predetermined timing. In some embodiments, thispredetermined timing may be sufficient to enable tracking 124 and/orobject recognition 122 while capture frame 502 remains relativelyunnoticed by a human eye.

FIG. 6 is a flowchart illustrating a process for projecting content andcapturing an image, consistent with some embodiments. For the purpose ofillustration, FIG. 6 will be described with reference to any of FIGS. 1,2A-2E, 3A-3B, 4A-4B, and 5A-5B. The process 600 shown in FIG. 6 may beembodied in computer-readable instructions in memory component 108,projection generation 120, object recognition 122, and/or tracking 124for execution by one or more processors therein or in processingcomponent 106 of device 100. In some embodiments, process 600 orportions thereof may be implemented by an operating system of device 100and/or may be implemented as a background service in the operatingsystem or subsystem of a component of device 100. As shown in FIG. 6,process 600 begins by projecting content at a first rate (602). In someembodiments, projection generation 120 based on object recognition 122,tracking 124, and user input may generate content for projection. Thecontent may be generated for projection onto one or more objects and/ortargets. In some embodiments, the content may be projected to augmentthe one or more objects and/or targets to create an augmented realityaround the one or more objects and/or targets. The content may be one ormore images, image frames, a series of image frames, or a combinationthereof. The first rate may be a projection rate between frame rate ofabout 23.976-300 frames per second (fps). In some embodiments, the firstrate may be about 72 fps.

Process 600 continues by projecting a capture frame at a second rate(604). In some embodiments, the capture frame may be a blank frame whereno content is projected, or a frame of a solid neutral color that isprojected on the field of view to allow image capture device to capturean image that does not obscure lines, features, or boundaries of objectswithin the field of view of image capture device 116 that may beprocessed for object recognition 122 or for tracking 124. Moreover, thecapture frame may include or be composed of a frame of high lightintensity that may be used to illuminate the field of view to facilitateimage capture by image capture device 116 in low-light situations. Thecapture frame may also be a frame that includes a fiducial or markerthat provides an indication that the image captured that includes thefiducial or marker is to be processed for performing computer visionfunctions. In some embodiments, the second rate that capture frame isprojected at may be a frame rate that is lower than the first framerate. In some embodiments, the second rate may be about 10-20 fps. Forexample, the second rate may be about 15 fps. In general, the secondrate is a rate that is sufficient for image capture device 116 tocapture an image without the capture frame being visible to the humaneye of a user of device 100 or an observer.

Process 600 may continue by determining if a capture frame is beingprojected (606). Determination of whether a capture frame is beingprojected may be made by receiving a signal from projection device 118that a capture frame is being projected. Alternatively, the projectionof a capture frame by projection device 118 may be synchronizedaccording to a clock signal generated by clock signal generator suchthat a capture frame is projected every x ms. Moreover, the projectionof a capture frame by projection device 118 may be synchronized based onthe number of content frames projected by projection device 118. Forexample, projection device 118 may be capable of projecting a captureframe every y number of content frames. Furthermore, process 600 maydetermine if a capture frame is being projected based on the presence orabsence of a fiducial or marker in a captured image.

If a capture frame is not being projected, projection device 118 isprojecting content at the first rate (602) and may continue projectingcontent until a capture frame is to be projected. When a capture frameis projected, image capture device 116 may capture an image of an areaincluding the field of view of image capture device 116 (608). Althoughprocess 600 describes not capturing an image of an area including thefield of view of image capture device 116 when a capture frame is notprojected, image capture device 116 may capture images at other times.For example, image capture device 116 may be configured to captureimages at the second rate that is lower than the first rate, but at arate than is different from the rate at which a capture frame isprojected. Based on a fiducial or marker on the capture frame, atimestamp of the capture frame, and the like, device 100 may be able todetermine which of the captured images to process for computer visionfunctions. Image capture device 116 may be configured to capture imagesat the first rate or a higher rate and determine which of the capturedimages to process for computer vision functions.

FIG. 7 is a flowchart illustrating a process for alternating projectingcontent and capturing an image, consistent with some embodiments. Forthe purpose of illustration, FIG. 7 will be described with reference toany of FIGS. 1, 2A-2E, 3A-3B, 4A-4B, and 5A-5B. The process 700 shown inFIG. 7 may be embodied in computer-readable instructions in memorycomponent 108, projection generation 120, object recognition 122, and/ortracking 124 for execution by one or more processors therein or inprocessing component 106 of device 100. In some embodiments, process 700or portions thereof may be implemented by an operating system of device100 and/or may be implemented as a background service in the operatingsystem or subsystem of a component of device 100. For example, the imagecapture and/or computer vision portions of process 700 may beimplemented in a camera subsystem, graphics processor, or video frontend (VFE) of device 100. As shown in FIG. 7, process 700 begins bycapturing an initial image (702). In some embodiments, the initial imagemay be captured by image capture device 116. Computer vision functionsmay then be performed on the captured image (704) to generate contentfor projection (706). For example, on an initial image objectrecognition 122 may be performed and on subsequent captured imagestracking 124 may be performed. Tracking 124 may include determining alocation and orientation of objects in a target area over a period oftime for directing content for projection in the target area and ontoone or more objects in the target area. In some embodiments, tracking124 may include estimating pose, orientation, and depth informationabout the target area and objects in the target area, such as object 1200 and/or object 2 300. Object recognition 122 may include performingone or more object recognition algorithms on the initial image and/orsubsequent captured images captured by image capture device 116 fortracking 124 the subsequent captured images and projection generation120. In some embodiments, object recognition 122 may include determiningor finding content associated with objects identified in the capturedinitial image.

Content may then be projected at a first rate (708). The content may begenerated for projection onto one or more objects and/or targets. Insome embodiments, the content may be projected to augment the one ormore objects and/or targets to create an augmented reality around theone or more objects and/or targets. The content may be one or moreimages, image frames, a series of image frames, or a combinationthereof. The first rate may be a projection rate between frame rate ofabout 23.976-300 frames per second (fps). In some embodiments, the firstrate may be about 72 fps. A capture frame may then be projected at asecond rate (710). In some embodiments, the capture frame may be a blankframe where no content is projected, or a frame of a solid neutral colorthat is projected on the field of view to allow image capture device tocapture an image that does not obscure lines, features, or boundaries ofobjects within the field of view of image capture device 116 that may beprocessed for object recognition 122 and/or for tracking 124. Moreover,the capture frame may include or be composed of a frame of high lightintensity that may be used to illuminate the field of view to facilitateimage capture by image capture device 116 in low-light situations. Insome embodiments, the capture frame may include a fiducial or markerthat may server as an indication that the image captured that includesthe fiducial or marker is to be processed for computer vision functions.In some embodiments, the second rate that capture frame is projected atmay be a frame rate that is lower than the first frame rate. In someembodiments, the second rate may be about 10-20 fps. For example, thesecond rate may be about 15 fps. In general, the second rate is a ratethat is sufficient for image capture device 116 to capture an imagewithout the capture frame being visible to the human eye of a user ofdevice 100 or an observer.

Process 700 may continue by determining if a capture frame is beingprojected (712). Determination of whether a capture frame is beingprojected may be made by receiving a signal from projection device 118that a capture frame is being projected. Alternatively, the projectionof a capture frame by projection device 118 may be synchronizedaccording to a clock signal generated by clock signal generator suchthat a capture frame is projected every x ms. Moreover, the projectionof a capture frame by projection device 118 may be synchronized based onthe number of content frames projected by projection device 118. Forexample, projection device 118 may be capable of projecting a captureframe every y number of content frames.

If a capture frame is not being projected, projection device 118 may beprojecting content at the first rate (708) and may continue projectingcontent until a capture frame is to be projected. When a capture frameis projected, image capture device 116 may capture an image of an areaincluding the field of view of image capture device 116 (714). Objectrecognition 122 and/or tracking 124 and/or other computer visionfunctions may then be performed on the captured image (704) to generatecontent for projection (706). In some embodiments, generating thecontent for projection may include generating the content for projectionbased on object recognition 122 and/or tracking 124. For example, if anobject has moved from its position in the initial captured image,tracking 124 may identify the new position of the object and projectiongeneration 120 may include projecting the content at the new position.Moreover, if an additional object has been included in a field of viewthat was not recognized in the initial image, object recognition 122 mayrecognize the new object and a position or orientation of the new objectmay be determined such that the projection generation 120 may includegenerating content based on the new object that may be projected ontoand around the new object. Process 700 may be capable of creating anaugmented reality on and around an object or a plurality of objectsdetected by object recognition 122. Moreover, process 700 may be used invideo games or other applications.

Although process 700 describes not capturing an image of an areaincluding the field of view of image capture device 116 when a captureframe is not projected, image capture device 116 may capture images atother times. For example, image capture device 116 may be configured tocapture images at the second rate that is lower than the first rate, butat a rate than is different from the rate at which a capture frame isprojected. Based on a fiducial or marker on the capture frame, atimestamp of the capture frame, and the like, device 100 may be able todetermine which of the captured images to process for computer visionfunctions. Image capture device 116 may be configured to capture imagesat the first rate or a higher rate and determine which of the capturedimages to process for computer vision functions.

Software, in accordance with the present disclosure, such as programcode and/or data, may be stored on one or more machine readable mediums,including non-transitory machine readable medium. It is alsocontemplated that software identified herein may be implemented usingone or more general purpose or specific purpose computers and/orcomputer systems, networked and/or otherwise. Where applicable, theordering of various steps described herein may be changed, combined intocomposite steps, and/or separated into sub-steps to provide featuresdescribed herein.

Consequently, embodiments as described herein may provide a system andmethod for alternating the projection of content and the capturing of animage during a period when no image is being projected. The embodimentsand examples provided herein may provide an improvement by allowingimage capture when no content is being projected so that the projectionmay be corrected or changed based on an image of an area that is notoccluded by the projection. The examples provided above are exemplaryonly and are not intended to be limiting. One skilled in the art mayreadily devise other systems consistent with the disclosed embodimentswhich are intended to be within the scope of this disclosure. As such,the application is limited only by the following claims.

What is claimed is:
 1. A method, comprising: projecting, by a projectiondevice, content at a first rate; projecting, by the projection device, acapture frame at a second rate; capturing, by an image capture device,an image when the capture frame is projected; and performing a computervision function based at least in part on the captured image.
 2. Themethod of claim 1, wherein projecting a capture frame comprisesprojecting one of a frame of a solid color and a frame having no image.3. The method of claim 1, wherein projecting the capture frame at thesecond rate comprises projecting the capture frame at the second ratesuch that the projected capture frame is not noticeable by an observer.4. The method of claim 1, wherein performing a computer vision functioncomprises: performing object recognition based on the captured image; orperforming tracking based on the captured image.
 5. The method of claim4, further comprising: generating the content for projection based onthe performed object recognition or tracking.
 6. The method of claim 4,wherein projecting content comprises projecting content that occludes anobject on which object recognition or tracking is performed.
 7. Themethod of claim 4, wherein performing tracking on the captured imagecomprises determining at least one of a position, orientation, and depthof an object in the captured image.
 8. The method of claim 1, wherein animage is captured at the second rate, and the method comprises:receiving, by the image capture device, a signal indicating that thecapture frame is being projected; and capturing the image in response toreceiving the signal.
 9. The method of claim 1, wherein an image iscaptured at a rate that is at least two times the second rate and themethod comprises synchronizing the projection device and the imagecapture device to capture at least one image including the capture frameand capture at least one image not including the capture frame.
 10. Asystem, comprising: a projection device configured to project content ata first rate and a capture frame at a second rate; an image capturedevice configured to capture an image when the projection deviceprojects the capture frame; one or more processors coupled to theprojection device and the image capture device, the one or moreprocessors configured to process the captured image and generate thecontent or determine a location or orientation of the content forprojection based at least in part on the captured image; and a memorycoupled to the one or more processors.
 11. The system of claim 10,wherein the projected capture frame comprises one of a solid-coloredcapture frame, and a capture frame having no image.
 12. The system ofclaim 10, wherein the image capture device comprises a camera.
 13. Thesystem of claim 12, wherein the camera comprises at least one of avisible light camera, a depth-sensing camera, a camera capable ofdetecting infrared (IR) light or ultraviolet (UV) light, a stereocamera, and a time-of-flight (ToF) camera.
 14. The system of claim 10,wherein the projection device comprises at least one of a Digital LightProcessing (DLP) projector, a laser beam-steering (LBS) projector, and aliquid crystal on silicon (LCoS) projector.
 15. The system of claim 10,wherein the one or more processors are configured to control theprojected content and a position of the projected content based on oneor more objects in the captured image.
 16. The system of claim 10,wherein the one or more processors are configured to perform objectrecognition on the captured image.
 17. The system of claim 10, whereinthe one or more processors are configured to perform tracking based onthe captured image.
 18. The system of claim 10, wherein the projectedcontent creates an augmented reality on or around one or more objects inthe captured image.
 19. A system, comprising: means for projectingcontent at a first rate; means for projecting a capture frame at asecond rate; and means for capturing an image when the capture frame isprojected; and means for performing a computer vision function based atleast in part on the captured image.
 20. The system of claim 19, whereinthe means for projecting a capture frame is configured to project atleast one of a frame of a solid color and a frame having no image. 21.The system of claim 19, wherein means for projecting the capture frameis configured to project the capture frame at the second rate such thatthe projected capture frame is not noticeable by an observer.
 22. Thesystem of claim 19, wherein the means for performing a computer visionfunction comprises at least one of: means for performing objectrecognition based on the captured image; and means for performingtracking based on the captured image.
 23. The system of claim 22,further comprising: means for generating the content for projectionbased on the performed object recognition or tracking.
 24. The system ofclaim 23, wherein the generated content for projection occludes anobject when projected, the object comprising an object on which theobject recognition or tracking is performed.
 25. The system of claim 22,wherein the means for performing tracking is configured to determine atleast one of a position, orientation, and depth of objects recognized inthe captured image.
 26. The system of claim 19, wherein the means forcapturing an image at the second rate comprises: means for receiving asignal indicating that the capture frame is being projected.
 27. Thesystem of claim 19, wherein the means for capturing an image and themeans for projecting are synchronized to capture an image at a thirdrate that is at least two times the second rate such that at least oneimage including the capture frame is captured and at least one image notincluding the capture frame is captured by the means for capturingduring operation of the system.
 28. A non-transitory computer-readablemedium including instructions that when executed by one or moreprocessors cause the one or more processors to perform a method,comprising: projecting content at a first rate; projecting a captureframe at a second rate; and capturing an image when the capture frame isprojected; and performing a computer vision function based at least inpart on the captured image.